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ABSTRACT 

Will computers lhal see and understand what they see revolutionize industry by 
automating the part orientation and pari inspection processes? There pre two 
obstacles; the expense ot computing and our feeble understanding of images. We. 
believe these obstacles are fast enditlf- To i Must rat* what can be done ws 
deserts a workini program that visually determines the position and orientation of 
silicon chips used in integrated circuits. 
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Will computers- that see and understand what they see revolutionize industry by 
automating the part orientation and part inspection processes? Predictions have 
been made and hedged, yet v#ry Hltle has been done so far. There are two 
reasons: 

* Computing power si ill costs too much. 

* Our understanding of real image* is too feeble. 

We believe these obstacles are erodin| quickly. There is row every indication 
thai difficult operations tan be automated soon with dramatic productivity 
increases. For one thing, powerlul computers are available now an single 
integrated circuit chips. At the same time new ideas have emerged that will 
enable machines to fill jobs that are too tedious or dangerous to attract human 
labor. Already one machine in the field it reading hand-written ZIP codes [I]. 
Another is visually inspecting printed- circuit cards for etching defects at faster 
than human speed and better than human reliability [2], 

To illustrate what can be dane> we describe a working program that 
visually determines the position and orientation of tiny silicon chips, a necessary 
prerequisite to establishing eteclrrcal contact between tire chips and the lead? 
protruding frpm the plastic cases enclosing them. Human operators perform this 
task for every transistor and integrated circuit made today by peering through a 
microscope or viewing a TSJ enlargement. 
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HOW INTEGRATED CIRCUITS ARE MADE 

Figure I 5 how* a typical intuited circuit <l C J layout and ji t „ re 2 ir| U5lrates th& 
way il would be creeled up inside a duef inline package fDI P ) The 
photcgrapba in figure* 3 end 4 show , C .' s lead-bonded into D.f.P/s and hybrid 
orcuits. Many dividual integrated circuit. are f*br, ca ted i oga th er n a »r> B i B 
two- ar three-inch wafer as i r, figure 5. As if it were- a sheet of fi asSp the wafer 
ii B «red with a diamond tool and then sp tit up into individual circuits ready ,„ 
■nstallatien in the standard plastic packages. Tiny pr.be electrode, ere applied to 
■ach chip in turn lo see if it W[J rks. Defective chip* era marked with an ink fipo l. 

After a chip I, bonded to the substrate automatic equipment can attach 
gold or aluminium wires from pad* on the chip to the much larger conductor that 
extend out of the package Unfortunate^ the target pad* are not registered 
precisely with the lines of chmH 9 nd the chip, s | ip a bit as they are i«d down, 
thus los,ng precise alignment. Consequently the position an d orientation of the 
chip must be detained accurately before [he automatic equipment lake, over 
Typically the human operator mo^es crosshairs tc the larget centers using a joy 
•lick. 

Newer techniques limit the problem somewhat by using \C. chips with 
reinforced gold-bell contacts as shown in figure 6. These can be bonded directly 
to either (enframes, as rn figure 7 or film-carrier 5| as in figure B, The greater 
exper.se of this process is warranted only because it simplifies the mam*] 
alignment Operation. Since the frame bear, fl prong for each pad, only one 
alignment operation i s needed for each chip, rather than one for each pad 
connected to. Still alignment remains the only operation not yet automated The 
««t of this is difficult to separate from other step,, but it ranges from perhaps 3 
«nts for simple chip, mass-produced abroad lo 30 cents for tomplicated chips 
made domestically. 
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A BORDER SIMPLIFIES PROCESSING 

Quick processing of complicated objects usually requires explanation of special 

properties thai make complete image analysis unnecessary. Sometimes slight 
redesign helps. In particular, addition of B n aluminium border to integrated circuit 
chips considerably simplifies I he orientation task. By happy accident some of the 
newer Schottky TTL chips now appear with such borders anyway, a s seen in 
figures 9 and 10. Since some chip area is sacrificed, generalizing our method may 
be desirable when handling chips which do not need borders for ether reasons. 

THE IMAGE CAN BE PROCESSED QUICKLY BY INEXPENSIVE EQUIPMENT 

Figure H i* a diagram of the system used. All modules are standard items; 
nothing is exotic Since the overall system was designed for flexible study of a 
broad range of applied problems, the sophistication and cost far exceed what in 
necessary. We estimate that a system coating 515,000 would do the job easily 
now, with further reduction likely as the cost of mini- and micro-computers 
continue* down {In fact there is a nice positive feedback loop at work since lead- 
bonding is a substantial component of the cost. of the integrated circuits inside the 
necessary processing equipment.) In addition, part of the equipment could be 
shared among a number of work-stations. 

Inexpensive or nut, no slow technique would be of much interest to 
industry. It is therefore fortunate that the visual processing can be done in less 
than .25 seconds. To do so requires some work, however. The mismatch of TV 
vidicon speed and computer speed is the first problem to face- Since a binary 
approximation ol the image cannot do the job, several hits of analog to digital 
conversion are needed. But multiple gray level conversion at video speed 
requires talented and expensive H to D hardware as well as fancy memory 
buffering. 

Picking off and cnnvf-rlhig one point per horizontal scan is the easy 
solution. Using this technique, inlensilies from one verlir.ai Ine of points can be 
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collects for eac h complete « t ef hoH2&nla | st9ns . T(ie antire . ms|e ^ ^ 
acquired by inching the t-m* between initiation f horizentar scar, and sampling, 
thus stepping the vertical line of sampled pmnt« across (he image. About 17 
seconds *r e required to record a complele i"ia|e of 480, by 512 pixels this way 
Fortunately, however, the chip orienting program need net lock at a complete 
image. In fact only those poinls Seated in figure 12 are qyite adequate and they 

are obta.nable from lo ur picture frames in 066 second. The lour diagona nes 

intersect the chip frontiers si eight points Finding them enables measurements of 
pad position accurate to .5 ™f Sf given that initial errors are wrtnin approximately 

Ax < 15 mil., Ay < 10 rnil h At < 25' 

Initial placement ordinarily &h within these limits. If the chip is further a w «,y from 
the expected position, nothing is losl beyond the time required for *, B *tr* pass 
af the process following coarse adjustment by the first try, There is a trada-off 
between accuracy and tolerance for large positioning errors, controlled by the 
placement of the tour diagonal scan lines. The positional accuracy of ,5 mile is 
quite satisfactory since the tenet pads are typically four mils en a side and th* 
connecting wire is one mil in diameter. 



THE IMAGE IS NOT BINARY 

Surprises await the person that thinks I.C. images ere trivial. | n , ml they are 
competed enough to be intare^ng but not so complicated as to require great 
sophistication. Under proper uniform lighting conditio^ the silicon materia? itself is 
relatively da* and the realization fighter and not very grainy, but Irensitioni 
between dark and light are not strictly step-like. Figure 13 illustrates the point. 
The intensity proves exhibit local irregularities rather than steady change because 
the edges f I.e. chips ere ragged and sloped like those ol mesas. Specurar 
refled-on points riddle these edges foiling reliance on simple threading. To do 
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the job therefore requires built-in knowledge about edge profile proteasing. 
Figure 5 illustrates these important steps: 

* Search lorward looking, (or the chip's aluminium border 
The first sign is a rapid rise in intensity extending over 
several pixels. 

* Next move further ahead looking lor the maximum intensity, 

* Search backward to find the minimum. 

+ Calculate [mm + minJ/3 and find where the waveform 

crosses this threshhold. This is (he hypothesized edge of the 
I.C 

The minimum and maximum points tend to be in the midst of plateaus from which 
they differ only slightly in intensity value. Consequently a small amount of noise 
somewhere on a plateau can move an extremum considerably, For this reason, 
averaging between the places wnere the minimum and maximum are found is not e 
good way to find the edge. Using the threshhold idea is a standard way to avoid 
the problem. 

Si gnat noise, dust, and scratches cause occasional miscalculation of an 
intersection point. This rarely interferes with correct positioning, however, 
because once the boundary is identified at above, bgi|t-in verification steps insure 
that the result is sound. These include checks on the difference between 
maximum and minimum, Ihe width ol the light and dark plateaus, the width of the 
transition, and so on, as illustrated by the measurements indicated in figure 14. A 
failure initiates further profile analysis and if this too (alls, the chip is moved a 
short distance in the best -guess direction and reseanfied. Geometrical tests ere 
also performed to ensure that the eight transilion points found do indeed lie on a 
rectangle and thai the rectangle has the correct dimensions. 
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Notice that individual rnle^ity pallet may deviate Considerably from 
th= expected shape. Some, for axaupie, may have an abrupt depression in the 
h-gh ,nl m5 ,ty plateau „ used by a nWma | gap ■„ lhfi border ^^ ^^ ^ 

sharp pinnacles where W^boidal fractures cause specular reflatl™. Template 
matching schem es are hard pressed by such problem. because it is hard to supply 
them w.lh specific knowledge about particular anomalies. Thankfully, biding 
knowledge inlc procedures is immensely easier. 

Altogether the approach resembles Shirt's treatment of scene analysia 
m the * or ld D f blocks and wedges, long a Wile of worker, In artificial 
-nteHigence [3J. indeed Shirks tenure point program fc a generally important 
source of ideas with which everyone in image preceding should be lami^r. 

CALIBRATION IS EASY 

The program reads the relationship between pixels and t he step size of the 
positioning table. There are several factors which influence this: pixel size v ar i es 
With each vicficon lube; the pixef 5ize differs in x and y, there is some relate 
rotation between the camera and Ihe UUe; ar*d the x and y axes in the camera 
are not exaclly orthogonal. The fO»owin 8 linear transformation account for these 
f pc tors: 



y. 



Yt 



Where « v and y v are pixel units and x, end y ( are table step units. 

Matrix elements a, b, c, and d are easily obtained by observing the displacement in 
the vpdienn which result* from .teppinj a standard I.C through known calibration 
por^tiL The inverse malrh, then gives carrecL labia movement for observed 

offsets. Refinement of this calibration process aimed at compensating for n on - 
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linear distortion In (he -■'idkon camera would increase accuracy, although further 
improvement does not seem necessary at the moment. 

AN INDUSTRIAL REVOLUTION IS NEEDED 

We believe Ihis application and others like it are the vanguard of b new kind of 
productivity technology,, on* oriented toward facing diflicull industrial vision and 
manipulation tasks squarely. Past solutions always involved using people or costly, 
inflexible special purpose automation, But people are moving increasingly toward 
the service -sector and away from the boredom o! the assembly line- Many of the 
things they do are difficult to automate because specif purpose automation costs 
too much or cannot work. Thus the need for progress in vision-based productivity 
technology is acute- 
Even in the electronics industry, well known far it* aggressive posture, 
with respect to new ideas, there are many other examples of expensive, manual 
assembly, inspection, and repair operations. Here are just a tew: 

Inspection of Printed Circuit Cards; 

When a large circuit hoard comes, out of the etching bath it is 
likely to have many crack and bridging defects These must 
be found and corrected before component insertion, Later on 
the board must be inspected again after wave soldering, 
k again with many hand COireCtiOns necessary, The Japanese 
have made some progress on this probrem [2\ 



Assembly of Electronic Circuits; 

Not all components can be handled by automatic insertion 
machines. Vtsual inspection is required in any case. Large or 
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odd- shaped components such as transformers and components 
with (loppy leads present special problems 



Repair ot Assembled Modules: 

With larger board areas the yield of working circuits h low. 
The "new" computer is actually a repaired computer 
debugged by an expensive, trained technician. 



Eventually machine vision is likely to irwreise productivity in such diverse 
activities, as undersea and underground mining, farming chores, surveillance of 

dangerous machines, package sorting, custodial care, planetary exploration, and the 
manufacture of shoes, clothes, and other consumer goods. Everyone knowi thai 
these are lou S h problem* to hand over to flexible computer-based systems. Until 
fww the technology has not been adequate and computation cost? have not been 
few enough to move machines with these capabilities out of the laboratory and into 
the manulacturing process. The great slrfdes that have been made In automating 
continuous processes tike oil refining and paper making, or in discrete parts 
manufacturing for (hat matter, hawe yet to be matched in situations requiring 
mechanical assembly or human quality visual inspection, It is time to catch up. 
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Figure 2i Sketch of tvpfcal Si^teen-pin dual - i ti 1 [ ne-packege E I lustratFng 
h&w ttle fnt*grit*d cfrcuFt cMp Is c&nmected to the lead& 
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Figure 3: Typical integrated circuit chips lead-bonded into fourteen--?! n 
dual- f nl f ne-packagcs using one-roM aluminium wire- 





Figure 4; Typfcai integrated cirquFt chTps lead-bonded Tnto hybrTd circuit- 
using one-mil gold wire. 
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Figure 6^ Chips with reinforced 
pads for lead-frame attachments 
(They happen to be defective ones) 



Figure 5' Standard 3" wafer, Only defective 
Chips have been left behind. 





FEgura 7: Laad-ff^mt strip of ten-fflSl 
gold-plated brass. 



FfcuJi-e 9; Fi3m-carrier strip. 





Ffflure «; rHcroflraph and phot* of T* monitor linage of Int^rated cfrcurt 
chip with bright aluntFnlun hardcf. Wot* dark gap in tb e border. 
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Figure 10; Myograph *r,d photo of TV mutter [mage of r nt « g r.ted circuit 

Chip »|, h bright aluminium border. N 0Ce M 9 h-lfgh» On pU 
caused by probing electrodes. 
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Figure 11: Gloete-di 5-gram of System used for laboratory demonst rai: fori of 

ariion-atic positioning and orient! rig of integrated circuit cblps 
using visual 1nput f 
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Figure 12; The four diagonal scan lines s. up C f imposed on a sketch of the 
image of <sn integrated circuit chrp. Circle? correspond to 
the eight edge points found by the program. 
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Figure 1J: II lust rat \oa of the waveform and measurertents taken to determine 
a hypothesized cd^e posi Lion- 




el . 



n\ 



Figure )h: I I I jst^at Ion of the waveform and some of the measu reiatn 1 5 taken 
to verify f hypothesized edge pcvltion- 



